{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Many to One RNN with Variable Sequence Length:\n",
    "\n",
    "In this tutorial we implement \n",
    "\n",
    "<img src=\"files/files/06.png\">\n",
    "\n",
    "\n",
    "*Fig1. Unfolded representation of the implemented RNN structure*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. Import the required libraries:\n",
    "We will start with importing the required libraries to our Python environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "ImportError",
     "evalue": "No module named tensorflow",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mImportError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-2df283633649>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# imports\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0mtensorflow\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mtf\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mnumpy\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mImportError\u001b[0m: No module named tensorflow"
     ]
    }
   ],
   "source": [
    "# imports\n",
    "import tensorflow as tf\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Generate some data\n",
    "\n",
    "For this tutorial ...\n",
    "\n",
    "### 1.1. Data dimension\n",
    "Here, we specify the dimensions of the data samples which will be used in the code. Defining these variables makes it easier (compared with using hard-coded number all throughout the code) to modify them later. Ideally these would be inferred from the data that has been read, but here we just write the numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Data Dimensions\n",
    "input_dim = 1           # input dimension\n",
    "seq_max_len = 4         # sequence maximum length\n",
    "out_dim = 1             # output dimension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Generate data and display the sizes\n",
    "Now we can use the defined helper function in \"train\" mode which loads the train and validation images and their corresponding labels. We'll also display their sizes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generate_data(count=1000, max_length=4, dim=1):\n",
    "    x = np.random.randint(0, 10, size=(count, max_length, dim))\n",
    "    length = np.random.randint(1, max_length+1, count)\n",
    "    for i in range(count):\n",
    "        x[i, length[i]:, :] = 0\n",
    "    y = np.sum(x, axis=1)\n",
    "    return x, y, length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x_train, y_train, seq_len_train = generate_data(count=1000, max_length=seq_max_len, dim=input_dim)\n",
    "x_test, y_test, seq_len_test = generate_data(count=5, max_length=seq_max_len, dim=input_dim)\n",
    "\n",
    "print(\"Size of:\")\n",
    "print(\"- Training-set:\\t\\t{}\".format(len(y_train)))\n",
    "print(\"- Test-set:\\t{}\".format(len(y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get batches of samples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def next_batch(x, y, seq_len, batch_size):\n",
    "    N = x.shape[0]\n",
    "    batch_indeces = np.random.permutation(N)[:batch_size]\n",
    "    x_batch = x[batch_indeces]\n",
    "    y_batch = y[batch_indeces]\n",
    "    seq_len_batch = seq_len[batch_indeces]\n",
    "    return x_batch, y_batch, seq_len_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Parameters\n",
    "learning_rate = 0.01    # The optimization initial learning rate\n",
    "training_steps = 10000  # Total number of training steps\n",
    "batch_size = 10         # batch size\n",
    "display_freq = 1000     # Frequency of displaying the training results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "learning_rate = 0.001 # The optimization initial learning rate\n",
    "epochs = 10           # Total number of training epochs\n",
    "batch_size = 100      # Training batch size\n",
    "display_freq = 100    # Frequency of displaying the training results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Network configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_hidden_units = 10   # number of hidden units"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Create network helper functions \n",
    "### 4.1. Helper functions for creating new variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# weight and bais wrappers\n",
    "def weight_variable(shape):\n",
    "    \"\"\"\n",
    "    Create a weight variable with appropriate initialization\n",
    "    :param name: weight name\n",
    "    :param shape: weight shape\n",
    "    :return: initialized weight variable\n",
    "    \"\"\"\n",
    "    initer = tf.truncated_normal_initializer(stddev=0.01)\n",
    "    return tf.get_variable('W',\n",
    "                           dtype=tf.float32,\n",
    "                           shape=shape,\n",
    "                           initializer=initer)\n",
    "\n",
    "\n",
    "def bias_variable(shape):\n",
    "    \"\"\"\n",
    "    Create a bias variable with appropriate initialization\n",
    "    :param name: bias variable name\n",
    "    :param shape: bias variable shape\n",
    "    :return: initialized bias variable\n",
    "    \"\"\"\n",
    "    initial = tf.constant(0., shape=shape, dtype=tf.float32)\n",
    "    return tf.get_variable('b',\n",
    "                           dtype=tf.float32,\n",
    "                           initializer=initial)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2. Helper-function for creating a RNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def RNN(x, weights, biases, n_hidden, seq_max_len, seq_len):\n",
    "    \"\"\"\n",
    "    :param x: inputs of shape [batch_size, max_time, input_dim]\n",
    "    :param weights: matrix of fully-connected output layer weights\n",
    "    :param biases: vector of fully-connected output layer biases\n",
    "    :param n_hidden: number of hidden units\n",
    "    :param seq_max_len: sequence maximum length\n",
    "    :param seq_len: length of each sequence of shape [batch_size,]\n",
    "    \"\"\"\n",
    "    cell = tf.nn.rnn_cell.BasicRNNCell(n_hidden)\n",
    "    outputs, states = tf.nn.dynamic_rnn(cell, x, sequence_length=seq_len, dtype=tf.float32)\n",
    "\n",
    "    # Hack to build the indexing and retrieve the right output.\n",
    "    batch_size = tf.shape(outputs)[0]\n",
    "    # Start indices for each sample\n",
    "    index = tf.range(0, batch_size) * seq_max_len + (seq_len - 1)\n",
    "    # Indexing\n",
    "    outputs = tf.gather(tf.reshape(outputs, [-1, n_hidden]), index)\n",
    "    out = tf.matmul(outputs, weights) + biases\n",
    "    return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Create the network graph\n",
    "### 5.1. Placeholders for the inputs (x), sequence length (seqLen), and corresponding labels (y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Placeholders for inputs(x), input sequence lengths (seqLen) and outputs(y)\n",
    "x = tf.placeholder(tf.float32, [None, seq_max_len, input_dim])\n",
    "seqLen = tf.placeholder(tf.int32, [None])\n",
    "y = tf.placeholder(tf.float32, [None, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2. Define the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create weight matrix initialized randomly from N~(0, 0.01)\n",
    "W = weight_variable(shape=[num_hidden_units, out_dim])\n",
    "\n",
    "# create bias vector initialized as zero\n",
    "b = bias_variable(shape=[out_dim])\n",
    "\n",
    "# Network predictions\n",
    "pred_out = RNN(x, W, b, num_hidden_units, seq_max_len, seqLen)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3. Define the loss function and optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define the loss function (i.e. mean-squared error loss) and optimizer\n",
    "cost = tf.reduce_mean(tf.square(pred_out - y))\n",
    "train_op = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.4. Initialize all variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Creating the op for initializing all variables\n",
    "init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    print('----------Training---------')\n",
    "    for i in range(training_steps):\n",
    "        x_batch, y_batch, seq_len_batch = next_batch(x_train, y_train, seq_len_train, batch_size)\n",
    "        _, mse = sess.run([train_op, cost], feed_dict={x: x_batch, y: y_batch, seqLen: seq_len_batch})\n",
    "        if i % display_freq == 0:\n",
    "            print('Step {0:<6}, MSE={1:.4f}'.format(i, mse))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Test\n",
    "### 7.1. Helper functions for plotting the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "    # Test\n",
    "    y_pred = sess.run(pred_out, feed_dict={x: x_test, seqLen: seq_len_test})\n",
    "    print('--------Test Results-------')\n",
    "    for i, x in enumerate(y_test):\n",
    "        print(\"When the ground truth output is {}, the model thinks it is {}\"\n",
    "              .format(y_test[i], y_pred[i]))\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
